Efficiency Analysis of Defect - Detection Techniques

Various effectiveness and efficiency metrics have been proposed for defect-detection techniques and quality assurance. This report aims at introducing and comparing the most common metrics that include the effort for the techniques. These metrics are based on code coverage and fault count. Furthermore two new metrics are introduced that use the failure intensity as a more reliability-oriented measure. The latter three metrics for determining efficiency are applied in a field study with the German software and system house ESG. Defect and test data from a three-year project is used to analyse the efficiency of the used techniques during four releases. The analysis showed that the efficiency increased at first but decreased in later releases. A correlation between the different metrics cannot be shown. Therefore only counting faults is not sufficient for measuring efficiency with respect to reliability. ∗This research was supported by the DFG within the priority program SoftSpez (SPP 1064) under project name InTime.